Skip to content

feat: add Jieba tokenizer for full-text search#111

Merged
lucasfang merged 14 commits into
alibaba:mainfrom
lxy-9602:fts-support-jieba
Feb 5, 2026
Merged

feat: add Jieba tokenizer for full-text search#111
lucasfang merged 14 commits into
alibaba:mainfrom
lxy-9602:fts-support-jieba

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Feb 3, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #69

  • Integrate cppjieba for full-text search
  • Support segmentation of mixed Chinese-English text
  • Add stop word filtering using jieba stop word list
  • Normalize English tokens to lowercase for case-insensitive matching
  • Support mix (default), mp, query, hmm, full tokenize mode
  • TODO: Skip stemming (e.g., Porter) for now;

Tests

JiebaAnalyzerTest
JiebaInterfaceTest
LuceneGlobalIndexTest, TestSimpleChinese

API and Format

lucene_defs.h: kJiebaTokenizeMode

Documentation

@lucasfang
lucasfang requested a review from Copilot February 4, 2026 06:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/paimon/global_index/lucene/lucene_api_test.cpp Outdated
Comment thread src/paimon/global_index/lucene/lucene_api_test.cpp Outdated
Comment thread src/paimon/global_index/lucene/lucene_global_index_writer.h Outdated
Comment thread src/paimon/global_index/lucene/jieba_analyzer.cpp Outdated
@alibaba alibaba deleted a comment from Copilot AI Feb 4, 2026
@alibaba alibaba deleted a comment from Copilot AI Feb 4, 2026
@alibaba alibaba deleted a comment from Copilot AI Feb 4, 2026
@alibaba alibaba deleted a comment from Copilot AI Feb 4, 2026
Comment thread src/paimon/global_index/lucene/lucene_global_index_reader.cpp
lucasfang
lucasfang previously approved these changes Feb 5, 2026

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lucasfang
lucasfang merged commit e8b2dcb into alibaba:main Feb 5, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants